Coresets For Monotonic Functions with Applications to Deep Learning

Coreset (or core-set) in this paper is a small weighted subset Q of the input set P with respect to a given monotonic function f : R → R that provably approximates its fitting loss ∑ p∈P f(p · x) to any given x ∈ R. Using Q we can obtain approximation to x∗ that minimizes this loss, by running existing optimization algorithms on Q. We provide: (i) a lower bound that proves that there are sets with no coresets smaller than n = |P | , (ii) a proof that a small coreset of size near-logarithmic in n exists for any input P , under natural assumption that holds e.g. for logistic regression and the sigmoid activation function. (iii) a generic algorithm that computes Q in O(nd+n logn) expected time, (iv) novel technique for improving existing deep networks using such coresets, (v) extensive experimental results with open code.